System and method for multiple-frame based super resolution interpolation for digital cameras

ABSTRACT

A digital camera system for super resolution image processing constructed to receive a plurality of input frames and output at least one digitally zoomed frame is provided. The digital camera system includes a motion registration module configured to generate motion information associated with the plurality of input frames, an interpolation module configured to generate a plurality of interpolated input frames based at least in part on the plurality of input frames and the motion information, a weights calculation module configured to calculate one or more weights associated with the plurality of input frames based on at least the motion information, and a weighted merging module configured to merge the plurality interpolated input frames consistent with the one or more weights to generate the at least one digitally zoomed frame.

BACKGROUND

1. Technical Field

The present invention relates to systems and methods for multiple-framebased super resolution interpolation for digital cameras.

2. Discussion

Digital cameras use various systems to enable photographers to captureimages of objects at long distances. Optical zoom systems use one ormore zoom lenses that may be adjusted to narrow the field of visibleradiation incident on the photo-detectors of the digital camera. Thenarrower field of visible radiation incident on the photo-detectorsmagnifies the captured image, albeit with a narrower field of view,without the introduction of significant image aberrations. In contrast,digital zoom systems increase the resolution of images, or a subset ofthe images, to create an effect similar to optical zoom (i.e., amagnified narrower field of view) through various image processingtechniques. Digital zoom systems, however, generally introducesignificant undesirable image aberrations relative to optical zoomsystems. For example, digital zoom systems may introduce aliasing (i.e.,jagged diagonal edges), blurring, and/or haloing into the image.

SUMMARY OF INVENTION

In accordance with at least one aspect of the embodiments disclosedherein, it is recognized that the standard approach to digital zoomsystems produces significant image aberrations including, but notlimited to, aliasing, blurring, and/or haloing. To minimize theintroduction of these image aberrations through digital zoom, systemsand methods for multiple-frame based super resolution interpolation fordigital cameras are provided. The systems and methods for superresolution digitally zooms input frames (i.e., images in a timesequence) by increasing the resolution of each frame throughinterpolation and merging the interpolated input frames consistent witha set of weights. For example, the set of weights may include one ormore weights for each pixel in each interpolated input frame. In thisexample, each pixel in the output digitally zoomed image is calculatedbased on a linear or non-linear combination of pixel values and theirassociated weights from the interpolated input frames.

According to one aspect, a digital camera system for super resolutionimage processing constructed to receive a plurality of input frames andoutput at least one digitally zoomed frame is provided. The digitalcamera system comprises a motion registration module configured togenerate motion information associated with the plurality of inputframes, an interpolation module configured to generate a plurality ofinterpolated input frames based at least in part on the plurality ofinput frames and the motion information, a weights calculation moduleconfigured to calculate one or more weights associated with theplurality of input frames based on at least the motion information, anda weighted merging module configured to merge the plurality interpolatedinput frames consistent with the one or more weights to generate the atleast one digitally zoomed frame.

According to one embodiment, the weighted merging module is furtherconfigured to calculate a weighted median of the interpolated inputframes based on the one or more weights.

According to one embodiment, the weights calculation module includes aBayer-pattern sensitive weights calculation module configured to computeat least one of the one or more frame weights based on a Bayer-patternconfiguration of a plurality of photoreceptors in an image sensor of thedigital camera system. According to one embodiment, the weightscalculation module further includes a registration confidence estimationmodule configured to compute at least one of the one or more weightsbased on an estimated accuracy level of the motion information.According to one embodiment, the weights calculation module furtherincludes a detail level estimation module configured to compute at leastone of the one or more frame weights based on a level of detail in atleast one of the plurality of frames.

According to one embodiment, the weights calculation module furtherincludes a registration confidence estimation module configured tocompute at least one of the one or more weights based on an estimatedaccuracy level of the motion information. According to one embodiment,the weights calculation module further includes a detail levelestimation module configured to compute at least one of the one or moreframe weights based on a level of detail in at least one of theplurality of frames.

According to one embodiment, the plurality of input frames includes atleast a current input frame and one of a previous input frame and aprevious output frame. According to one embodiment, the weighted mergingmodule is further configured to generate a frame weight map includingweight information based on the one or more weights. According to oneembodiment, the weights calculation module includes a weight mapinterpolation module configured to compute at least one of the one ormore weights based on the frame weight map of a previous output frame.

According to one embodiment, the plurality of input frames have a firstframe rate and the at least one digitally zoomed frame has a secondframe rate that is less than the first rate.

According to one aspect, a method of super resolution image processingin a digital camera is provided. The method comprises receiving aplurality of input frames, generating motion information associated withthe plurality of input frames, interpolating the plurality of inputframes based on the plurality of input frames and the motion informationto generate a plurality of interpolated input frames, calculating one ormore weights associated with the plurality of input frames based on atleast the motion information, and merging the plurality interpolatedinput frames consistent with the on the one or more weights associatedwith the plurality of frames to generate at least one digitally zoomedframe.

According to one embodiment, merging the plurality of interpolated inputframes includes calculating a weighted median of the interpolated inputframes based on the one or more weights.

According to one embodiment, calculating the one or more weightsincludes calculating at least one of the one or more weights based on aBayer-pattern configuration of a plurality of photoreceptors in an imagesensor of the digital camera system. According to one embodiment,calculating the one or more weights further includes calculating atleast one of the one or more weights based on an estimated accuracylevel of the motion information. According to one embodiment,calculating the one or more weights further includes calculating atleast one of the one or more weights based on a level of detail in atleast one of the plurality of input frames.

According to one embodiment, receiving the plurality of input framesincludes receiving at least a current input frame and one of a previousinput frame and a previous output frame. According to one embodiment,merging the plurality interpolated input frames may include generating aframe weight map including weight information based on the one or moreweights. According to another embodiment, calculating the one or moreweights may include calculating at least one of the one or more weightsbased on the frame weight map of a previous frame.

According to one aspect, a non-transitory computer readable mediumhaving stored thereon sequences of instructions for super resolutionimage processing is provided. The instructions include instructions thatinstruct at least one processor to receive a plurality of input frames,generate motion information associated with the plurality of inputframes, interpolate the plurality of input frames based on the pluralityof input frames and the motion information to generate a plurality ofinterpolated input frames, calculate one or more weights associated withthe plurality of input frames based on at least the motion informationand a level of detail in at least one of the plurality of frames, andmerge the plurality interpolated input frames consistent with the on theone or more weights associated with the plurality of frames to generateat least one digitally zoomed frame.

Still other aspects, embodiments and advantages of these exemplaryaspects and embodiments, are discussed in detail below. Moreover, it isto be understood that both the foregoing information and the followingdetailed description are merely illustrative examples of various aspectsand embodiments, and are intended to provide an overview or frameworkfor understanding the nature and character of the claimed aspects andembodiments. Any embodiment disclosed herein may be combined with anyother embodiment. References to “an embodiment,” “an example,” “someembodiments,” “some examples,” “an alternate embodiment,” “variousembodiments,” “one embodiment,” “at least one embodiment,” “this andother embodiments” or the like are not necessarily mutually exclusiveand are intended to indicate that a particular feature, structure, orcharacteristic described in connection with the embodiment may beincluded in at least one embodiment. The appearances of such termsherein are not necessarily all referring to the same embodiment.

BRIEF DESCRIPTION OF THE DRAWINGS

Various aspects of at least one example are discussed below withreference to the accompanying figures, which are not intended to bedrawn to scale. The figures are included to provide an illustration anda further understanding of the various aspects and examples, and areincorporated in and constitute a part of this specification, but are notintended as a definition of the limits of the embodiments disclosedherein. The drawings, together with the remainder of the specification,serve to explain principles and operations of the described and claimedaspects and examples. In the figures, each identical or nearly identicalcomponent that is illustrated in various figures is represented by alike numeral. For purposes of clarity, not every component may belabeled in every figure. In the figures:

FIG. 1 illustrates a functional block diagram of a super resolutionimage processing system in accordance with the present invention;

FIG. 2 illustrates an image processing system in which super resolutionimage processing may be performed;

FIG. 3 illustrates a more detailed functional block diagram of a superresolution image processing system for super resolution image processingin accordance with an embodiment of the present invention; and

FIG. 4 illustrates a more detailed functional block diagram of a superresolution image processing system for super resolution image processingin accordance with another embodiment of the present invention.

DETAILED DESCRIPTION

It is to be appreciated that examples of the methods and apparatusesdiscussed herein are not limited in application to the details ofconstruction and the arrangement of components set forth in thefollowing description or illustrated in the accompanying drawings. Themethods and apparatuses are capable of implementation in other examplesand of being practiced or of being carried out in various ways. Examplesof specific implementations are provided herein for illustrativepurposes only and are not intended to be limiting. In particular, acts,elements and features discussed in connection with any one or moreexamples are not intended to be excluded from a similar role in anyother examples.

Also, the phraseology and terminology used herein is for the purpose ofdescription and should not be regarded as limiting. Any references toexamples or elements or acts of the systems and methods herein referredto in the singular may also embrace examples including a plurality ofthese elements, and any references in plural to any example or elementor act herein may also embrace examples including only a single element.References in the singular or plural form are not intended to limit thepresently disclosed systems or methods, their components, acts, orelements. The use herein of “including,” “comprising,” “having,”“containing,” “involving,” and variations thereof is meant to encompassthe items listed thereafter and equivalents thereof as well asadditional items. References to “or” may be construed as inclusive sothat any terms described using “or” may indicate any of a single, morethan one, and all of the described terms.

Embodiments of the present invention relate to multiple-frame basedsuper resolution interpolation for digital cameras. It should beappreciated that the term “digital camera” used herein includes, but isnot limited to, dedicated cameras, video cameras, as well as camerafunctionality performed by any electronic device (e.g., mobile phones,personal digital assistants, etc.). In addition, the use herein of theterm “module” is interchangeable with the term “block” and is meant toinclude implementations of processes and/or functions in software,hardware, or a combination of software and hardware. In one embodiment,the multiple-frame based super resolution interpolation systems andmethods digitally zoom input frames, or any portion of the input frames,without introducing significant aberrations into the output digitallyzoomed frames. Digitally zooming images typically introduces significantaberrations including, for example, aliasing, blurring, and/or haloing.Accordingly, in some embodiments, the systems and methods formultiple-frame based super resolution interpolation reduce noise inaddition to increasing detail in digitally zoomed frames.

Super Resolution Image Processing System

FIG. 1 illustrates an example system for super resolution imageprocessing. As shown, the super resolution image processing system 100of FIG. 1 includes a super resolution engine 102 that receives inputframes 104 and performs image processing on the received input frames104 to produce at least one digitally zoomed frame 106. The superresolution engine 102 includes multiple functional blocks or modulesincluding a motion registration block 108 to determine motioninformation associated with the input frames 104, an illuminationadjustment block 109 to compensate for illumination changes in the inputframes 104, an interpolation block 110 to increase the resolution of theinput frames 104, a weight calculation block 112 to determine one ormore weights to combine the input frames 104, and a weighted mergingblock 114 to combine the input frames 104 into the digitally zoomedframe 106 consistent with the one or more weights.

In various embodiments, the input frames 104 include images arranged in,for example, a time sequence or an encoder sequence. Each imagecomprises a plurality of pixels represented by one or more twodimensional matrices where each element in the matrices representsinformation regarding a specific pixel location in the image. Forexample, the location (10, 10) in a first, second, and thirdtwo-dimensional matrix may represent data associated with a pixel at the(10, 10) location in the image. Each of the one or more matrices mayrepresent a set of intensities associated with the image in a givendimension. In one embodiment, each pixel in the image is represented bya three-dimensional vector. In this embodiment, the image is representedby a first two dimensional matrix for the first dimension, a secondtwo-dimensional matrix for the second dimension, and a thirdtwo-dimensional matrix for the third dimension. It is appreciated thateach pixel may be represented by any number of dimensions or any numberof matrices.

In some embodiments, the image is represented by three two-dimensionalmatrices consistent with a YUV color space. The YUV color space iscomposed of three distinct components Y, U, and V where each twodimensional matrix represents a specific component of the YUV space. TheY component is the luminance component that relates to the brightness ofthe image. The U and V components are chrominance components that relateto the specific color of the image. Each pixel in an image isrepresented by a vector in the YUV color space (i.e., some combinationof the Y, U, and V components). In some embodiments, the image isrepresented by three two-dimensional matrices consistent with a RGBcolor space. The RGB color space is also composed of three distinctcomponents R, G, and B where each two-dimensional matrix represents aspecific component of the RGB space. All three of the distinctcomponents (i.e., R, G, and B components) are all chrominance componentsthat relate to the specific color of the image. It is appreciated thatthe image may be represented in any color space and is not limited tothe YUV or RGB color spaces.

The super resolution engine 102 executes a variety of processes tooutput the at least one digitally zoomed frame 106 based on the inputframes 104. The various processes performed by the super resolutionengine 102 to generate the digitally zoomed frame 106 include theprocesses performed by the motion registration block 108, theillumination adjustment block 109, the interpolation block 110, theweight calculation block, and the weighted merging block 114.

In some embodiments, the motion registration module 108 determinesmotion information associated with the input frames 104. For example,the motion registration module 108 may generate a series of motionvectors representative of the trajectory of pixels in the input frames104. The motion information generated by the motion registration module108 may be utilized by the interpolation module 110 in conjunction withthe input frames 104 to generate interpolated input frames. Theinterpolation module 110 increases the resolution of the input frames104 by one or more interpolation processes including, for example,bicubic interpolation, bilinear interpolation, nearest neighborinterpolation, spline interpolation, or sinc interpolation. Theinterpolation block 110 may be preceded by an illumination adjustmentblock 109 to compensate for global and/or local illumination changesbetween the input frames 104 prior to interpolation. The illuminationadjustment block 109 may, for example, compute a difference betweenlow-pass filtered input frames.

The interpolated input frames are merged by the weighted merging module114 to produce the output digitally zoomed frame 106 consistent with oneor more weights generated by the weight calculation block 112. Theweight calculation block 112 determines the one or more weights based onvarious image information including, for example, motion information,Bayer-pattern information, and image detail level information.

In one embodiment, the time intervals between the input frames 104 issmaller than the time intervals between the digitally zoomed outputframes 106. For example, a digital camera may capture multiple images ina time sequence of an object and the super resolution engine 102 mayoutput a single still frame. In another example, the super resolutionengine 102 may receive input frames 104 at a frame rate in excess of 30frames per second and output digitally zoomed frames at a frame rate of30 frames per second.

In various embodiments, the output digitally zoomed frame 106 isprocessed by one or more single-frame based image processing processesto further improve the quality of the image. Such single-frame basedimage processing processes are described in co-pending application Ser.No. 13/921,712, titled SYSTEM AND METHOD FOR SINGLE-FRAME BASED SUPERRESOLUTION INTERPOLATION FOR DIGITAL CAMERAS, filed on Jun. 19, 2013(hereinafter the “'712 application”) which is incorporated by referenceherein in its entirety. For example, the edge enhancement moduledisclosed in the '712 application may be employed in a single-frame postprocessing process to improve the quality of object edges. The edgeenhancement module disclosed in the '712 application improves thequality of object edges in an image by extracting the edges of objectsin the image, optionally enhancing the extracted edges, and combiningthe extracted edges with the input image.

It is appreciated that any combination of the super resolution imageprocessing functional blocks (e.g., the motion registration block 108,the illumination adjustment block 109, the interpolation block 110, theweight calculation block 112, and the weighted merging block 114) mayperform operations in parallel. In one embodiment, the input frames 104may be divided into a plurality of subsections. Each subsection of theinput frames may be processed individually in parallel. The processedsubsections may be combined to form the output digitally zoomed frame106. Implementing the super resolution image processing in parallelincreases the speed in which the digitally zoomed frame 106 can beprovided.

It is appreciated that the super resolution engine 102 may take avariety of forms dependent upon the specific application. In someembodiments, the super resolution engine 102 comprises a series ofinstructions performed by an image processing system. In otherembodiments, one or more of the blocks or modules 108, 109, 110, 112,and 114 may be implemented in hardware, such as an application-specificintegrated circuit (ASIC), system on a chip (SoC), state machine, or adedicated processor.

Example Image Processing System

FIG. 2 illustrates an example image processing system 200 in which superresolution processing can be performed. As shown, the image processingsystem 200 of FIG. 2 includes a digital camera 202 that is configured toprovide digital images and/or video of a scene 218. The digital camera202 includes an image processor 204, an analog-to-digital converter 206,a memory 208, an interface 210, storage 212, an image sensor 214, and alens 216.

As illustrated in FIG. 2, the digital camera 202 includes the imageprocessor 204 to implement at least some of the aspects, functions andprocesses disclosed herein. The image processor 204 performs a series ofinstructions that result in manipulated data. The image processor 204may be any type of processor, multiprocessor or controller. Someexemplary processors include processors with ARM11 or ARM9architectures. The image processor 204 is connected to other systemcomponents, including one or more memory devices 208.

The memory 208 stores programs and data during operation of the digitalcamera 202. Thus, the memory 208 may be a relatively high performance,volatile, random access memory such as a dynamic random access memory(“DRAM”) or static memory (“SRAM”). However, the memory 208 may includeany device for storing data, such as a flash memory or othernon-volatile storage devices. Various examples may organize the memory208 into particularized and, in some cases, unique structures to performthe functions disclosed herein. These data structures may be sized andorganized to store values for particular data and types of data.

The data storage element 212 includes a writeable nonvolatile, ornon-transitory, data storage medium in which instructions are storedthat define a program or other object that is executed by the imageprocessor 204. The data storage element 212 also may include informationthat is recorded, on or in, the medium, and that is processed by theimage processor 204 during execution of the program. More specifically,the information may be stored in one or more data structuresspecifically configured to conserve storage space or increase dataexchange performance. The instructions may be persistently stored asencoded signals, and the instructions may cause the image processor 204to perform any of the functions described herein. The medium may, forexample, be optical disk, magnetic disk or flash memory, among others.In operation, the image processor 204 or some other controller causesdata to be read from the nonvolatile recording medium into anothermemory, such as the memory 208, that allows for faster access to theinformation by the image processor 204 than does the storage mediumincluded in the data storage element 212. The memory may be located inthe data storage element 212 or in the memory 208, however, the imageprocessor 204 manipulates the data within the memory, and then copiesthe data to the storage medium associated with the data storage element212 after processing is completed. A variety of components may managedata movement between the storage medium and other memory elements andexamples are not limited to particular data management components.Further, examples are not limited to a particular memory system or datastorage system.

The digital camera 202 also includes one or more interface devices 210such as input devices, output devices and combination input/outputdevices. Interface devices may receive input or provide output. Moreparticularly, output devices may render information for externalpresentation. Input devices may accept information from externalsources. Examples of interface devices include microphones, touchscreens, display screens, speakers, buttons, printers, etc. Interfacedevices allow the digital camera 202 to exchange information and tocommunicate with external entities, such as users and other systems.

Although the digital camera 202 is shown by way of example as one typeof digital camera upon which various aspects and functions may bepracticed, aspects and functions are not limited to being implemented onthe digital camera 202 as shown in FIG. 2. Various aspects and functionsmay be practiced on digital cameras having a different architectures orcomponents than that shown in FIG. 2. For instance, the digital camera202 may include specially programmed, special-purpose hardware, such asan application-specific integrated circuit (“ASIC”) or a system on achip (“SoC”) tailored to perform a particular operation disclosedherein. It is appreciated that the digital camera 202 may beincorporated into another electronic device (e.g., mobile phone,personal digital assistant etc.) and is not limited to dedicated digitalcameras or video cameras.

The lens 216 includes one or more lenses that focus the visibleradiation on the image sensor 214. It is appreciated that the lens 216is not limited to a single physical lens as illustrated in FIG. 2. Insome embodiments, the lens 216 includes a plurality of zoom lenses thatenable optical zoom. Optical zoom may be accomplished by narrowing thefield of view of the visible radiation incident on the image sensor 214.

The image sensor 214 may include a two-dimensional area of sensors(e.g., photo-detectors) that are sensitive to light. In someembodiments, the photo-detectors of the image sensor 214, in someembodiments, can detect the intensity of the visible radiation in one oftwo or more individual color and/or brightness components. For example,the output of the photo-detectors may include values consistent with aYUV or RGB color space. It is appreciated that other color spaces may beemployed by the image sensor 214 to represent the captured image and thephoto-detectors may be arranged consistent with one or more patterns.For example, the photo-detectors may be arranged consistent with a Bayerpattern. In this example, the photo-detector grid is constructed todetect green light at 50% of the photo-detectors, red light at 25% ofthe photo-detectors, and blue light at 25% of the photo-detectors.

In various embodiments, the image sensor 214 outputs an analog signalproportional to the intensity and/or color of visible radiation strikingthe photo-detectors of the image sensor 214. The analog signal output bythe image sensor 214 may be converted to digital data by theanalog-to-digital converter 206 for processing by the image processor204. In some embodiments, the functionality of the analog-to-digitalconverter 206 is integrated with the image sensor 214. The imageprocessor 204 may perform a variety of processes to the captured imagesand/or video. These processes may include, but are not limited to, oneor more super resolution processes to digitally zoom the capturedimages.

Super Resolution Image Processing Implementations

FIG. 3 illustrates a more detailed functional block diagram of the superresolution image processing system 100 depicted in FIG. 1. As shown inFIG. 3, the super resolution image processing system 300 of FIG. 3receives input frames 302 and processes the received input frames 302though a motion registration block 108, an illumination adjustment block109, an interpolation block 110, a weight calculation block 112, and aweighted merging block 114 to produce at least one digitally zoomedoutput frame 304. In accordance with one embodiment of the superresolution imaging processing system, the weight calculation block 112in the super resolution image processing system 300 includes a detaillevel estimation block 306, a Bayer-Pattern sensitive weights calculatorblock 308, and a registration confidence estimation block 310.

The motion registration block 108 generates motion information based onthe received input frames 302. The motion information includes, forexample, one or more matrices of motion vectors where each motion vectorrepresents the trajectory of a given pixel through the received inputframes 302. The motion information may be generated consistent with amotion model including, for example, an affine model and include localmotion correction and/or motion constraints. It is appreciated that theset of motion vectors are not limited to being associated with pixels inan image. For example, motion vectors may be associated with sections orsub-pixels of the input frames 302.

The interpolation block 110 increases the resolution of the receivedinput frames 302 through an interpolation process to generateinterpolated input frames. Example interpolation processes includebicubic interpolation, bilinear interpolation, nearest neighborinterpolation, spline interpolation, and sinc interpolation. It isappreciated that the interpolation block 110 may use motion informationin the interpolated process to improve the quality of the interpolatedinput frames. The interpolation block 110 may be preceded by anillumination adjustment block 109 to compensate for local and/or globalillumination changes between frames prior to interpolation.

The weight calculation block 112 determines one or more weightsassociated with the input frames 302 that are employed by the weightedmerging block 114 to combine the interpolated input frames. The detaillevel estimation block 308, the Bayer-Pattern sensitive weightscalculator block 310, and the registration confidence block 312 are oneillustration of the weight calculator block 112.

The detail level estimation block 306 calculates one or more weightsbased on an identified amount of detail in the input frames 302 and/orthe output frame 304. The detail level estimation block 306 reduces theweight of frames, or sections of frames, in the output frame 304 thathave motion-blur or focus changes that cause detail blurring. It isappreciated that the detail level estimation block 306 may identify thelevel of detail on a per-frame basis or a per-pixel basis.

The Bayer-pattern sensitive weights calculator 310 calculates one ormore weights based on the Bayer-pattern information from the inputframes 302 and motion information from the motion registration block108. As discussed above with reference to the image sensor 214 of imageprocessing system 200, an image sensor is comprised of a two-dimensionalarray of photo-detectors. The photo-detector grid may be arrangedconsistent with a Bayer-pattern and thereby detect green light at 50% ofthe photo-detectors, red light at 25% of the photo-detectors, and bluelight at 25% of the photo-detectors. The Bayer-pattern sensitive weightscalculator 310 takes into account the Bayer-pattern of the input frames302 in addition to the motion information to give more weight to regionsor pixels in the interpolated frame that are closer to their originatedsensor location (i.e., the sensor producing the pixel data). Conversely,the Bayer-pattern sensitive weights calculator 310 will give less weightto regions or pixels in the interpolated frame that are further awayfrom their originated sensor location.

The registration confidence estimation block 312 calculates one or moreweights based on the quality of the motion information. The registrationconfidence estimation block 312 identifies areas in the frame where thegenerated motion vector did not register well between the input frames302. The areas that registered poorly may include, for example, occludedand revealed regions in addition to regions whose motion is inconsistentwith an assumed motion model, if any motion model is assumed. Theregistration confidence estimation block 312 reduces the weight offrames, regions of frames, or pixels where the generated motioninformation quality is poor.

The weighted merging block 114 generates the output digitally zoomedframe 304 by merging the interpolated input images consistent with theone or more weights calculated by the weight calculation block 112. Theweighted merging block 114 may employ a function to merge theinterpolated frames including, for example, a weighted median of theinterpolated frames.

Having described at least one implementation of the super resolutionimage processing system, it is appreciated that the specificimplementations described may be altered to generate the same result ofdigitally zooming an image without introducing significant imageaberrations. Another example implementation of the super resolutionimage processing system is illustrated by functional block diagram ofsuper resolution image processing system 400 with reference to FIG. 4.

FIG. 4 illustrates an alternative detailed functional block diagram ofthe super resolution image processing system 100 depicted in FIG. 1. Asshown in FIG. 4, the super resolution image processing system 400 ofFIG. 4 receives a current input frame and a previous input frame 402, alast output frame 404, and a last frame weight map 418 and generates anoutput frame 414 in addition to a frame weight map 416. The superresolution image processing system 400 processes the received imageinformation through a motion registration block 108, an illuminationadjustment block 109, an interpolation block 110, a weight calculationblock 112, and a weighted merging block 114. In accordance with oneembodiment of the super resolution image processing system, the weightcalculation block 112 in the super resolution image processing system400 includes a detail level estimation block 406, a Bayer-Patternsensitive weights calculator block 408, a registration confidenceestimation block 410, and a weight map interpolation block 412.

In one embodiment, the super resolution image processing system 400reduces the required processing power by only merging two frames foreach output frame. The super resolution image processing system 400interpolates the current and last input frames 402 in the interpolationblock 110. The interpolated frames are merged in the weighted mergingblock 114 as described above with reference to FIGS. 1 and 3. Theimplementation in FIG. 4 reduces the requirements for processing poweras well as reducing memory size and bandwidth requirements from thesystem. The weighted merging block 114 further outputs a frame weightmap 416. The frame weight map 416 may include, for example, one or morematrices representative of the one or more weights or a combined metriccalculated by the weight calculation block 112. The frame weight map ofthe previous frame 418 is then employed by the weight map interpolationblock 412 within the weight calculation block 112 to generate one ormore weights for the current frame.

In some embodiments, the motion registration block 108 generates motioninformation based on the current input frame and one of the last inputframe and the last output frame 404. The motion information (e.g.,motion error information or similar) is employed by the weightcalculation block 112 as described above with reference to FIG. 3. It isappreciated that the detail level estimation block 406, theBayer-Pattern sensitive weights calculator block 408, and theregistration confidence estimation block 410 may, or may not, beconstructed to be identical to the detail level estimation block 306,the Bayer-Pattern sensitive weights calculator block 308, and theregistration confidence estimation block 310, respectively, as describedabove with reference to FIG. 3.

Having thus described several aspects of at least one example, it is tobe appreciated various alterations, modification, and improvements willreadily occur to those skilled in the art. Such alterations,modifications, and improvements are intended to be part of thisdisclosure, and are intended to be within the scope of the embodimentsdisclosed herein. Accordingly, the foregoing description and drawingsare by way of example only.

What is claimed is:
 1. A digital camera system for super resolutionimage processing constructed to receive a plurality of input frames andoutput at least one digitally zoomed frame, the digital camera systemcomprising at least one processor configured to: generate motioninformation associated with the plurality of input frames; generate aplurality of interpolated input frames based at least in part on theplurality of input frames and the motion information; calculate one ormore weights associated with the plurality of input frames based on atleast the motion information; and merge the plurality of interpolatedinput frames consistent with the one or more weights to generate the atleast one digitally zoomed frame and generate a frame weight map for thedigitally zoomed frame, the frame weight map including weightinformation based on the one or more weights.
 2. The digital camerasystem of claim 1, wherein the at least one processor is furtherconfigured to calculate a weighted median of the interpolated inputframes based on the one or more weights.
 3. The digital camera system ofclaim 1, wherein the at least one processor is further configured tocompute at least one of the one or more frame weights based on aBayer-pattern configuration of a plurality of photoreceptors in an imagesensor of the digital camera system.
 4. The digital camera system ofclaim 3, wherein the at least one processor is further configured tocompute at least one of the one or more weights based on an estimatedaccuracy level of the motion information.
 5. The digital camera systemof claim 4, wherein the at least one processor is further configured tocompute at least one of the one or more frame weights based on a levelof detail in at least one of the plurality of frames.
 6. The digitalcamera system of claim 1, wherein the at least one processor is furtherconfigured to compute at least one of the one or more weights based onan estimated accuracy level of the motion information.
 7. The digitalcamera system of claim 6, wherein the at least one processor is furtherconfigured to compute at least one of the one or more frame weightsbased on a level of detail in at least one of the plurality of frames.8. The digital camera system of claim 1, wherein the plurality of inputframes includes at least a current input frame and one of a previousinput frame and a previous output frame.
 9. The digital camera system ofclaim 1, wherein the at least one processor is further configured tocompute at least one of the one or more weights based on the frameweight map of a previous output frame.
 10. The digital camera system ofclaim 1, wherein the plurality of input frames have a first frame rateand the at least one digitally zoomed frame has a second frame rate thatis less than the first rate.
 11. A method of super resolution imageprocessing in a digital camera, the method comprising: receiving aplurality of input frames; generating motion information associated withthe plurality of input frames; interpolating the plurality of inputframes based on the plurality of input frames and the motion informationto generate a plurality of interpolated input frames; calculating one ormore weights associated with the plurality of input frames based on atleast the motion information; and merging the plurality of interpolatedinput frames consistent with the one or more weights associated with theplurality of frames to generate at least one digitally zoomed frame andgenerating a frame weight map for the digitally zoomed frame, the frameweight map including weight information based on the one or moreweights.
 12. The method of claim 11, wherein merging the plurality ofinterpolated input frames includes calculating a weighted median of theinterpolated input frames based on the one or more weights.
 13. Themethod of claim 11, wherein calculating the one or more weights includescalculating at least one of the one or more weights based on aBayer-pattern configuration of a plurality of photoreceptors in an imagesensor of the digital camera.
 14. The method of claim 13, whereincalculating the one or more weights further includes calculating atleast one of the one or more weights based on an estimated accuracylevel of the motion information.
 15. The method of claim 14, whereincalculating the one or more weights further includes calculating atleast one of the one or more weights based on a level of detail in atleast one of the plurality of input frames.
 16. The method of claim 11,wherein receiving the plurality of input frames includes receiving atleast a current input frame and one of a previous input frame and aprevious output frame.
 17. The method of claim 11, wherein calculatingthe one or more weights includes calculating at least one of the one ormore weights based on the frame weight map of a previous frame.
 18. Anon-transitory computer readable medium having stored thereon sequencesof instructions for super resolution image processing that instruct atleast one processor to: receive a plurality of input frames; generatemotion information associated with the plurality of input frames;interpolate the plurality of input frames based on the plurality ofinput frames and the motion information to generate a plurality ofinterpolated input frames; calculate one or more weights associated withthe plurality of input frames based on at least the motion informationand a level of detail in at least one of the plurality of frames; andmerge the plurality of interpolated input frames consistent with the oneor more weights associated with the plurality of frames to generate atleast one digitally zoomed frame and generate a frame weight map for thedigitally zoomed frame, the frame weight map including weightinformation based on the one or more weights.